Image sensor with near-infrared and visible light pixels

ABSTRACT

An image sensor may include an array of imaging pixels and an array of color filter elements that covers the array of imaging pixels. The array of imaging pixels may include visible light pixels that are covered by visible light color filter elements and near-infrared light pixels that are covered by near-infrared light color filter elements. The imaging pixels may be arranged in a pattern having a repeating 2×2 unit cell of pixel groups. Each pixel group may include a visible light pixel sub-group and a near-infrared light pixel sub-group. Signals from each pixel group may be processed to determine a representative value for each pixel group that includes both visible light and near-infrared light information.

This application is a continuation of application Ser. No. 15/783,022,filed Oct. 13, 2017, which claims the benefit of and claims priority toprovisional patent application No. 62/510,333, filed May 24, 2017, whichare hereby incorporated by reference herein in their entireties.

BACKGROUND

This application relates to image sensors, and more particularly, imagesensors that have visible and near-infrared pixels.

Image sensors are commonly used in electronic devices such as cellulartelephones, cameras, and computers to capture images. In a typicalarrangement, an electronic device is provided with an array of imagepixels arranged in pixel rows and pixel columns. The image pixelscontain a photodiode for generating charge in response to light.Circuitry is commonly coupled to each pixel column for reading out imagesignals from the image pixels. A color filter element typically coverseach photodiode.

Several image sensor applications (such as security cameras) requirevisible light and near-infrared (NIR) image sensor sensitivity at thesame time. Conventional systems use a physically moveable IR filter toobtain near-infrared and visible light sensitivity. However, this isimpractical and there is a strong need for a low-cost image sensor withboth visible light and near-infrared (NIR) sensitivity.

It would therefore be desirable to provide image sensors with visibleand near-infrared light sensitivity.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram of an illustrative electronic device having an imagesensor in accordance with an embodiment.

FIG. 2 is a diagram of an illustrative pixel array and associatedreadout circuitry for reading out image signals in an image sensor inaccordance with an embodiment.

FIG. 3 shows an illustrative image sensor with a color filter patternthat may be used to provide an image sensor with both visible light andnear-infrared sensitivity but that requires custom pattern processing inaccordance with an embodiment.

FIGS. 4-13 show illustrative image sensors with color filter patternsthat may be used to provide an image sensor with both visible light andnear-infrared sensitivity without requiring custom pattern processing inaccordance with an embodiment.

FIGS. 14 and 15 show illustrative image sensors with custom pixellayouts for visible light and near-infrared light sensitivity inaccordance with an embodiment.

FIGS. 16 and 17 are diagrams of illustrative method steps that may beused in processing signals from an image sensor with visible light andnear-infrared light sensitivity in accordance with an embodiment.

DETAILED DESCRIPTION

Electronic devices such as digital cameras, computers, cellulartelephones, and other electronic devices may include image sensors thatgather incoming light to capture an image. The image sensors may includearrays of pixels. The pixels in the image sensors may includephotosensitive elements such as photodiodes that convert the incominglight into image signals. Image sensors may have any number of pixels(e.g., hundreds or thousands or more). A typical image sensor may, forexample, have hundreds of thousands or millions of pixels (e.g.,megapixels). Image sensors may include control circuitry such ascircuitry for operating the pixels and readout circuitry for reading outimage signals corresponding to the electric charge generated by thephotosensitive elements.

FIG. 1 is a diagram of an illustrative imaging and response systemincluding an imaging system that uses an image sensor to capture images.System 100 of FIG. 1 may an electronic device such as a camera, acellular telephone, a video camera, or other electronic device thatcaptures digital image data, may be a vehicle safety system (e.g., anactive braking system or other vehicle safety system), or may be asurveillance system.

As shown in FIG. 1, system 100 may include an imaging system such asimaging system 10 and host subsystems such as host subsystem 20. Imagingsystem 10 may include camera module 12. Camera module 12 may include oneor more image sensors 14 and one or more lenses.

Each image sensor in camera module 12 may be identical or there may bedifferent types of image sensors in a given image sensor arrayintegrated circuit. During image capture operations, each lens may focuslight onto an associated image sensor 14. Image sensor 14 may includephotosensitive elements (i.e., pixels) that convert the light intodigital data. Image sensors may have any number of pixels (e.g.,hundreds, thousands, millions, or more). A typical image sensor may, forexample, have millions of pixels (e.g., megapixels). As examples, imagesensor 14 may include bias circuitry (e.g., source follower loadcircuits), sample and hold circuitry, correlated double sampling (CDS)circuitry, amplifier circuitry, analog-to-digital converter circuitry,data output circuitry, memory (e.g., buffer circuitry), addresscircuitry, etc.

Still and video image data from camera sensor 14 may be provided toimage processing and data formatting circuitry 16 via path 28. Imageprocessing and data formatting circuitry 16 may be used to perform imageprocessing functions such as data formatting, adjusting white balanceand exposure, implementing video image stabilization, face detection,etc. Image processing and data formatting circuitry 16 may also be usedto compress raw camera image files if desired (e.g., to JointPhotographic Experts Group or JPEG format). In a typical arrangement,which is sometimes referred to as a system on chip (SOC) arrangement,camera sensor 14 and image processing and data formatting circuitry 16are implemented on a common semiconductor substrate (e.g., a commonsilicon image sensor integrated circuit die). If desired, camera sensor14 and image processing circuitry 16 may be formed on separatesemiconductor substrates. For example, camera sensor 14 and imageprocessing circuitry 16 may be formed on separate substrates that havebeen stacked.

Imaging system 10 (e.g., image processing and data formatting circuitry16) may convey acquired image data to host subsystem 20 over path 18.Host subsystem 20 may include processing software for detecting objectsin images, detecting motion of objects between image frames, determiningdistances to objects in images, filtering or otherwise processing imagesprovided by imaging system 10.

If desired, system 100 may provide a user with numerous high-levelfunctions. In a computer or advanced cellular telephone, for example, auser may be provided with the ability to run user applications. Toimplement these functions, host subsystem 20 of system 100 may haveinput-output devices 22 such as keypads, input-output ports, joysticks,and displays and storage and processing circuitry 24. Storage andprocessing circuitry 24 may include volatile and nonvolatile memory(e.g., random-access memory, flash memory, hard drives, solid-statedrives, etc.). Storage and processing circuitry 24 may also includemicroprocessors, microcontrollers, digital signal processors,application specific integrated circuits, etc.

An example of an arrangement for camera module 12 of FIG. 1 is shown inFIG. 2. As shown in FIG. 2, camera module 12 includes image sensor 14and control and processing circuitry 44. Control and processingcircuitry 44 may correspond to image processing and data formattingcircuitry 16 in FIG. 1. Image sensor 14 may include a pixel array suchas array 32 of pixels 34 (sometimes referred to herein as image sensorpixels or image pixels 34). Control and processing circuitry 44 may becoupled to row control circuitry 40 and may be coupled to column controland readout circuitry 42 via data path 26. Row control circuitry 40 mayreceive row addresses from control and processing circuitry 44 and maysupply corresponding row control signals to image pixels 34 over controlpaths 36 (e.g., dual conversion gain control signals, pixel resetcontrol signals, charge transfer control signals, blooming controlsignals, row select control signals, or any other desired pixel controlsignals). Column control and readout circuitry 42 may be coupled to thecolumns of pixel array 32 via one or more conductive lines such ascolumn lines 38. Column lines 38 may be coupled to each column of imagepixels 34 in image pixel array 32 (e.g., each column of pixels may becoupled to a corresponding column line 38). Column lines 38 may be usedfor reading out image signals from image pixels 34 and for supplyingbias signals (e.g., bias currents or bias voltages) to image pixels 34.During image pixel readout operations, a pixel row in image pixel array32 may be selected using row control circuitry 40 and image dataassociated with image pixels 34 of that pixel row may be read out bycolumn control and readout circuitry 42 on column lines 38.

Column control and readout circuitry 42 may include column circuitrysuch as column amplifiers for amplifying signals read out from array 32,sample and hold circuitry for sampling and storing signals read out fromarray 32, analog-to-digital converter circuits for converting read outanalog signals to corresponding digital signals, and column memory forstoring the read out signals and any other desired data. Column controland readout circuitry 42 may output digital pixel values to control andprocessing circuitry 44 over line 26.

Array 32 may have any number of rows and columns. In general, the sizeof array 32 and the number of rows and columns in array 32 will dependon the particular implementation of image sensor 14. While rows andcolumns are generally described herein as being horizontal and vertical,respectively, rows and columns may refer to any grid-like structure(e.g., features described herein as rows may be arranged vertically andfeatures described herein as columns may be arranged horizontally).

If desired, array 32 may be part of a stacked-die arrangement in whichpixels 34 of array 32 are split between two or more stacked substrates.In such an arrangement, each of the pixels 34 in the array 32 may besplit between the two dies at any desired node within pixel. As anexample, a node such as the floating diffusion node may be formed acrosstwo dies. Pixel circuitry that includes the photodiode and the circuitrycoupled between the photodiode and the desired node (such as thefloating diffusion node, in the present example) may be formed on afirst die, and the remaining pixel circuitry may be formed on a seconddie. The desired node may be formed on (i.e., as a part of) a couplingstructure (such as a conductive pad, a micro-pad, a conductiveinterconnect structure, or a conductive via) that connects the two dies.Before the two dies are bonded, the coupling structure may have a firstportion on the first die and may have a second portion on the seconddie. The first die and the second die may be bonded to each other suchthat first portion of the coupling structure and the second portion ofthe coupling structure are bonded together and are electrically coupled.If desired, the first and second portions of the coupling structure maybe compression bonded to each other. However, this is merelyillustrative. If desired, the first and second portions of the couplingstructures formed on the respective first and second dies may be bondedtogether using any known metal-to-metal bonding technique, such assoldering or welding.

As mentioned above, the desired node in the pixel circuit that is splitacross the two dies may be a floating diffusion node. Alternatively, thenode between a floating diffusion region and the gate of a sourcefollower transistor (i.e., the floating diffusion node may be formed onthe first die on which the photodiode is formed, while the couplingstructure may connect the floating diffusion node to the source followertransistor on the second die), the node between a floating diffusionregion and a source-drain node of a transfer transistor (i.e., thefloating diffusion node may be formed on the second die on which thephotodiode is not located), the node between a source-drain node of asource-follower transistor and a row select transistor, or any otherdesired node of the pixel circuit.

It may be desirable for image sensor 14 to have both visible light andnear-infrared (NIR) sensitivity. Accordingly, image sensor 14 mayinclude both visible light color filter elements and near-infrared colorfilter elements over pixel array 32. FIG. 3 shows an illustrative imagesensor 14 with a color filter pattern that may be used to provide animage sensor with both visible light and near-infrared sensitivity.Color filter elements 52 may be formed over the pixel array in apattern. Each color filter element 52 may cover a corresponding pixel(34). Pixels with a green color filter element are labeled “G”, pixelswith a red color filter element are labeled “R”, pixels with a bluecolor filter element are labeled “B”, and pixels with a near-infraredcolor filter element are labeled “N.” The pattern of FIG. 3 is a 4×4unit cell that may be repeated across the image sensor. Although thepattern of FIG. 3 provides the image sensor with visible light and NIRsensitivity, the pattern of FIG. 3 may require custom pattern processing(because the output of image sensor 14 is a 4×4 unit square instead of a2×2 unit square Bayer color filter pattern).

FIGS. 4-13 show illustrative image sensors with color filter patternsthat may be used to provide an image sensor with both visible light andnear-infrared sensitivity. Pixels with a green color filter element arelabeled “G”, pixels with a red color filter element are labeled “R”,pixels with a blue color filter element are labeled “B”, and pixels witha near-infrared color filter element are labeled “N.” Pixels covered bya visible light (e.g., red, green, blue, etc.) color filter element maybe referred to as visible light pixels and pixels covered by anear-infrared color filter element may be referred to as near-infraredlight pixels. The patterns of FIGS. 4-13 may include 4×4 unit cells thatmay be repeated across the array of pixels in the imaging sensor. Thepatterns of FIGS. 4-13 may include strategically placed NIR pixels tominimize the amount of additional processing required to output a Bayerpattern. Since the red, green, and blue pixels are also sensitive to NIRlight, these can saturate under strong NIR light (i.e. incandescentillumination), significantly reducing the dynamic range and increasingthe noise even when they are not saturated. Processing may be done in away that detects problematic, high NIR content regions and desaturatesthese regions (providing monochrome high-fidelity luma output),minimizing visual artifacts.

In the color filter pattern of FIG. 4, fifty percent of the pixels areNIR pixels. As shown, the pattern of FIG. 4 may include two pixelsgrouped together in adjacent rows and a single column (sometimesreferred to as a 1×2 or 2×1 arrangement). These 1×2 groups of pixels maysometimes be referred to as sub-groups. For example, the pattern of FIG.4 may include sub-groups 54-1 (with two green color filter elements),54-2 (with two near-infrared color filter elements), 54-3 (with two redcolor filter elements), 54-4 (with two near-infrared color filterelements), 54-5 (with two blue color filter elements), 54-6 (with twonear-infrared color filter elements), 54-7 (with two green color filterelements), and 54-8 (with two near-infrared color filter elements). Thesub-groups may further form groups of pixels (and color filterelements). For example, sub-group 54-1 and sub-group 54-2 may form afirst group, sub-group 54-3 and sub-group 54-4 may form a second group,sub-group 54-5 and sub-group 54-6 may form a third group, and sub-group54-7 and sub-group 54-8 may form a fourth group. In other words, eachgroup of color filter elements may be a quadrant of the 4×4 unit cell.If desired, each quadrant of the 4×4 unit cell may include a sub-groupof near-infrared color filter elements and a sub-group of visible light(e.g., red, green, or blue) color filter elements.

In FIG. 4, the sub-group of visible light color filter elements in eachquadrant follows the Bayer color filter pattern (with green sub-groupsdiagonally opposite one another and a red sub-group that diagonallyopposes a blue sub-group). This may minimize the amount of processingrequired for an image sensor using the color filter pattern of FIG. 4 tooutput a Bayer-type pattern. For example, based on the color filterpattern in FIG. 4, each quadrant (or group) of pixels may be processedto obtain a value that includes visible light information andnear-infrared light information. Based on the 4×4 unit square of FIG. 4,the output of the image sensor may be values associated with a 2×2 unitsquare (e.g., like a Bayer pattern). For example, the upper-leftquadrant may have an output value associated with green lightinformation and near-infrared information, the upper-right quadrant mayhave an output value associated with red light information andnear-infrared information, the lower-left quadrant may have an outputvalue associated with blue light information and near-infraredinformation, and the lower-right quadrant may have an output valueassociated with green light information and near-infrared information.The processed output may then advantageously be compatible with Bayerpattern processing techniques.

The signals from both pixels in a sub-group may be summed or averagedduring processing. Each pixel may be covered by a respective microlens,or each sub-group of pixels may be covered by a respective singlemicrolens. In some embodiments, four pixels in a 2×2 square (e.g., agroup or quadrant of pixels) may be covered by a single microlens. Insome embodiments (such as FIG. 6), four pixels of the same type (e.g.,the two groups of 2×2 NIR pixels) may each be covered by a respectivemicrolens (with each microlens covering four pixels). In someembodiments (such as FIG. 4), 2×2 pixel groups with different types ofpixels may be covered by a single microlens. For example, in FIG. 4 thetwo green pixels and two NIR pixels in the upper left quadrant may becovered by a single microlens if desired.

In embodiments where multiple pixels are covered by a single microlens(i.e., a 2×1 pixel group covered by a single microlens or a 2×2 pixelgroup covered by a single microlens), the sensor may also be used forphase detection. The signal levels of each individual pixel may beconsidered to obtain phase detection information.

One possible advantage of the color filter pattern of FIG. 4 is that thepattern exhibits proper Bayer center-of-mass location. This may beadvantageous in improving performance of the image sensor. Other colorfilter patterns may be also used (as shown in FIGS. 5-15). The patternsof FIGS. 5-13 may also include 4×4 unit cells that may be repeatedacross the array of pixels in the imaging sensor. Each pattern of FIGS.5-13 may include both visible light color filter elements andnear-infrared color filter elements. For all of the color filterpatterns, processing may be performed to obtain a 2×2 unit square ofoutput, with each value of the 2×2 square including visible lightinformation and near-infrared light information. In the patterns ofFIGS. 4-9, fifty percent of the pixels may be near-infrared pixels. Inthe patterns of FIGS. 10-13, twenty-five percent of the pixels may benear-infrared pixels. As shown in FIGS. 4, 5, 7, and 8, each pixel group(e.g., quadrant) of the 4×4 unit square may include two visible lightpixels and two near-infrared pixels in various arrangements. In somearrangements as shown in FIGS. 6, 9, and 11, each pixel group (e.g.,quadrant) of the 4×4 unit square may include either all visible lightpixels or all near-infrared pixels. In some arrangements as shown inFIGS. 12 and 13, each pixel group (e.g., quadrant) of the 4×4 unitsquare may include may include three visible light pixels and onenear-infrared pixel.

FIGS. 4-13 all show embodiments where the image sensor has an array ofphotodiodes arranged in rows and columns (with each photodiode beingcovered by a respective color filter element). However, in some cases acustom pixel layout may be used as shown in FIGS. 14 and 15. In FIG. 14,photodiodes for NIR pixels are surrounded by photodiodes for visiblelight pixels. A single NIR pixel may be surrounded by a single visiblelight pixel in each quadrant of a 2×2 unit cell. This arrangement hasthe advantage of proper Bayer center-of-mass location. Similarly, FIG.15 shows an image sensor with four NIR pixels surrounded by four visiblelight pixels in each quadrant of a 2×2 unit cell. This arrangement alsohas the advantage of proper Bayer center-of-mass location. For both ofthe arrangements of FIGS. 14 and 15, processing may be performed toobtain a 2×2 unit square of output, with each value of the 2×2 squareincluding visible light information and near-infrared light information.

FIGS. 16 and 17 show methods used in processing signals from an imagesensor that includes visible light pixels and near-infrared lightpixels. FIG. 16 is a high-level diagram showing how signals from asensor with visible light pixels and near-infrared light pixels areprocessed. FIG. 16 may use the color filter pattern of FIG. 4, as anexample. However, these methods may be applied to any desired colorfilter pattern (e.g., any of the patterns of FIGS. 4-15). As shown inFIG. 16, each 4×4 unit square in the image sensor may be processedseparately. An illustrative 4×4 unit square 102 (of the type shown inFIG. 4) is processed in FIG. 16. During processing, the 4×4 unit square102 may be split into a processed near-infrared 2×2 unit square 104 anda processed visible light 2×2 unit square 106.

For example, processed near-infrared 2×2 unit square 104 may be based onthe signals from near-infrared pixels in 4×4 unit square 102. Forexample, signals from the near-infrared pixels of group 56-1 in 4×4 unitsquare 102 may be used to determine the processed value N′ in theupper-left quadrant of processed 2×2 unit square 104. Signals from thenear-infrared pixels of group 56-2 in 4×4 unit square 102 may be used todetermine the processed value N′ in the upper-right quadrant ofprocessed 2×2 unit square 104. Signals from the near-infrared pixels ofgroup 56-3 in 4×4 unit square 102 may be used to determine the processedvalue N′ in the lower-left quadrant of processed 2×2 unit square 104.Signals from the near-infrared pixels of group 56-4 in 4×4 unit square102 may be used to determine the processed value N′ in the lower-rightquadrant of processed 2×2 unit square 104. Additional processing may beperformed to obtain each N′ value (e.g., adjusting the near-infraredsignals based on information from neighboring visible light pixels).

Similarly, signals from the green pixels of group 56-1 in 4×4 unitsquare 102 may be used to determine the processed value G′ in theupper-left quadrant of processed 2×2 unit square 106. Signals from thered pixels of group 56-2 in 4×4 unit square 102 may be used to determinethe processed value R′ in the upper-right quadrant of processed 2×2 unitsquare 106. Signals from the blue pixels of group 56-3 in 4×4 unitsquare 102 may be used to determine the processed value B′ in thelower-left quadrant of processed 2×2 unit square 106. Signals from thegreen pixels of group 56-4 in 4×4 unit square 102 may be used todetermine the processed value G′ in the lower-right quadrant ofprocessed 2×2 unit square 106. Additional processing may be performed toobtain the R′, G′, and B′ values (e.g., adjusting the visible lightsignals based on information from neighboring near-infrared lightpixels).

After obtaining processed near-infrared 2×2 unit square 104 and aprocessed visible light 2×2 unit square 106, the processed 2×2 unitsquares 104 and 106 may be mixed at step 108. During mixing, the valuesof processed near-infrared 2×2 unit square 104 and the values ofprocessed visible light 2×2 unit square 106 may be combined using amixing scheme. Any desired mixing scheme may be used. In one example,the values of processed near-infrared 2×2 unit square 104 and the valuesof processed visible light 2×2 unit square 106 may be combined using aratio that is determined based on the amount of visible andnear-infrared light present. For example, if near-infrared light is veryhigh and visible light is very low, the values from processednear-infrared 2×2 unit square 104 will be given a high weight duringmixing and the values from processed visible light 2×2 unit square 106will be given a very low weight during mixing. In this scenario, thevalues from processed near-infrared 2×2 unit square 104 may be the finaloutput (e.g., 100% of the output may be from unit square 104 and 0% ofthe output may be from unit square 106). In contrast, if near-infraredlight is very low and visible light is very high, the values fromprocessed near-infrared 2×2 unit square 104 will be given a low weightduring mixing and the values from processed visible light 2×2 unitsquare 106 will be given a very high weight during mixing. In thisscenario, the values from processed visible light 2×2 unit square 106may be the final output (e.g., 100% of the output may be from unitsquare 106 and 0% of the output may be from unit square 104). In otherwords, during the mixing step, an interpolation of the values of squares104 and 106 may be taken (based on the visible and NIR light levels) andprovided as output. The interpolation may be done on a per value basis(e.g., each quadrant may be interpolated independently) or on a per unitsquare basis (e.g., each quadrant may be interpolated in the samemanner).

Mixing step 108 may be performed by circuitry within the system 100(e.g., image processing and data formatting circuitry 16, storage andprocessing circuitry 24, etc.). The circuitry that performs mixing step108 may be referred to as mixing circuitry. The mixing circuitry may mixmonochrome image data (from processed near-infrared 2×2 unit square 104)with Bayer image data (from processed visible light 2×2 unit square 106)in step 108. If desired, white balance gains may be applied to the Bayerimage data during mixing (e.g., to processed visible light 2×2 unitsquare 106). Alternatively, inverse white balance gains may be appliedto the monochrome image data during mixing (e.g., to processednear-infrared 2×2 unit square 104).

In this way, a single output image may capture both NIR portions andvisible light portions of a single scene. For example, the monochromeimage data may be used in dark portions of the output image (where thescene is only illuminated with NIR light) while other portions of thesame output image may use Bayer image data, creating a hybrid image thatprovides information regarding the scene in areas both with and withoutvisible light.

The output from the mixing step may undergo demosaicing at step 110.Because the data output from mixing step 108 is Bayer-type data, nocustomizations may need to be made to demosaicing step 110. Thedemosaicing step may be performed by a signal processor 112 (e.g., imageprocessing and data formatting circuitry 16, storage and processingcircuitry 24, or any other desired processing circuitry). Demosaiceddata (YUV data) may be output from processing circuitry 112.

FIG. 17 is a diagram of illustrative steps for processing image datafrom an image sensor with visible light pixels and near-infrared pixels.As shown, at step 202 processing circuitry (e.g., image processing anddata formatting circuitry 16 in FIG. 1) may analyze visible andnear-infrared light levels (e.g., from 4×4 unit square 102 in FIG. 16).One or more processing paths may then be chosen based on the visible andnear-infrared light levels (e.g., the processing path including steps204 and 206, the processing path including steps 208 and 210, theprocessing path including steps 212 and 214, and/or the processing pathincluding the steps 216 and 218). If desired, the signals from visibleand near-infrared light pixels in 4×4 unit square 102 may be compared tothresholds while determining the light levels. Processing the signalsbased on the light levels may help account for the visible light pixelsalso being sensitive to near-infrared light. Because the visible lightpixels are sensitive to near-infrared light, the signals from thevisible light pixels may include contributions from both visible andnear-infrared light. This may be accounted for during processing basedon the levels of both near-infrared and visible light.

If the visible light level is determined to be low (e.g., if a signalfrom one visible light pixel, the signals from all visible light pixels,and/or an average signal from two or more visible light pixels is belowa given threshold such as RGB_(LOW)), processing may proceed to step204. In step 204, the signals from each pixel in each group (e.g.,quadrant) of unit square 102 may be averaged to obtain a representativenear-infrared signal N′ for that group. For example, looking at FIG. 16,the signals from the four pixels in group 56-1 may be averaged to obtainN′ in the upper-left quadrant of unit square 104, the signals from thefour pixels in group 56-2 may be averaged to obtain N′ in theupper-right quadrant of unit square 104, etc. This may produce a 2×2grid of representative N′ values that is output at step 206.

If, as determined at step 202, the visible light level is not low (e.g.,if a signal from one visible light pixel, the signals from all visiblelight pixels, and/or an average signal from two or more visible lightpixels is above a given threshold such as RGB_(LOW)) and there is morenear-infrared light than visible light (e.g., NIR>RGB), processing mayproceed to step 208. In step 208, the signals from each near-infraredpixel in each group (e.g., quadrant) of unit square 102 may be averagedto obtain a representative near-infrared signal N′ for that group. Forexample, looking at FIG. 16, the signals from the two near-infraredpixels in group 56-1 may be averaged to obtain N′ in the upper-leftquadrant of unit square 104, the signals from the two near-infraredpixels in group 56-2 may be averaged to obtain N′ in the upper-rightquadrant of unit square 104, etc. This may produce a 2×2 grid ofrepresentative N′ values that is output at step 210.

If there is less near-infrared light than visible light (e.g., NIR<RGB),than processing may proceed from step 202 to step 212. At step 212, thesignals from each visible light pixel in each group may be averaged. Thesignals from near-infrared pixels that are adjacent to the visible lightpixels may then be averaged. The difference between the average visiblelight pixel signal and the average near-infrared light pixel signal maybe used as a representative signal (e.g., R′, G′, or B′) for that group.For example, consider group 56-2 in FIG. 16. First, the average of thesignals from the two red pixels in group 56-2 may be obtained. Next, theaverage of the signals from the four adjacent near-infrared pixels(e.g., the two NIR pixels in group 56-2 and the two NIR pixels in group56-1) may be obtained. The difference in the two averages may beobtained to determine R′ for the upper-right quadrant of unit square 106in FIG. 16. This process may be continued for each group of pixels toproduce a 2×2 grid of representative R′, G′, and B′ values (e.g.,processed visible light 2×2 unit square 106 in FIG. 16) that is outputat step 214.

If the near-infrared light levels are low (e.g., if a signal from onenear-infrared light pixel, the signals from all near-infrared lightpixels, and/or an average signal from two or more near-infrared lightpixels is below a given threshold such as NIR_(LOW)), processing mayproceed from step 202 to step 216. The signals from each visible lightpixel in each group may be averaged to obtain a representative signal(R′, G′, or B′) for that group. For example, the signals from the twogreen pixels in group 56-1 may be averaged to obtain G′ in theupper-left quadrant of unit square 106, the signals from the two redpixels in group 56-2 may be averaged to obtain R′ in the upper-rightquadrant of unit square 106, etc. The resulting 2×2 grid ofrepresentative R′, G′, and B′ values (e.g., processed visible light 2×2unit square 106 in FIG. 16) is output at step 218.

During step 220, the outputs from steps 206, 210, 214, and/or 218 may bemixed. Depending on the light levels determined in step 202, processingmay produce outputs from one, two, three, or four of the fourillustrative processing paths. The outputs from each of the processingpaths may be mixed at step 220 based on the light levels. For example,during mixing the processing circuitry may smoothly interpolate betweenany of the four possible outputs. If desired, white balance gains may beapplied to the RGB data during mixing (e.g., to the outputs from steps214 or 218). Alternatively, inverse white balance gains may be appliedto the NIR data during mixing (e.g., to the outputs from steps 206 or210) then white balance gains may be applied during subsequentprocessing.

As discussed above in connection with FIG. 16, circuitry may mixmonochrome image data (e.g., outputs from steps 206 and/or 210) withBayer image data (e.g., outputs from steps 214 and/or 218) at step 220.In this way, a single output image may capture both NIR portions andvisible light portions of a single scene. For example, the monochromeimage data may be used in dark portions of the output image (where thescene is only illuminated with NIR light) while other portions of thesame output image may use Bayer image data, creating a hybrid image thatprovides information regarding the scene in areas both with and withoutvisible light.

It should be noted that during any of the aforementioned processingsteps of FIG. 17, a low pass filter may be applied to the signals. Forexample, a low pass filter may be applied to the raw data from thepixels (e.g., the 4×4 unit square 102 of FIG. 16) or to the processed2×2 unit squares (e.g., the outputs from steps 206, 210, 214 and/or218). Similarly, all of the aforementioned processing steps may utilizeresampling to match spatial locations of the output.

Importantly, the output of mixing step 220 (which may the same as mixingstep 108 in FIG. 16) may be a Bayer pattern (e.g., a repeating 2×2 unitcell of values with two values including green light information, onevalue including red light information, and one value including bluelight information in each 2×2 unit cell). By enabling the processor(e.g., processor 112 in FIG. 16 that performs demosaicing) to receive aBayer pattern, the processor can be saved from having to perform custompattern processing.

In the aforementioned embodiments, various color filter patterns aredescribed as having visible light color filter elements andnear-infrared color filter elements. It should be noted that these colorfilter patterns are merely illustrative. Different color filter elementsmay be substituted for the visible light color filter elements ifdesired. For example, a clear color filter element may be used in placeof the green color filter elements if desired. Green color filterelements, red color filter elements, blue color filter elements, yellowcolor filter elements, cyan color filter elements, magenta color filterelements, broadband color filter elements (e.g., clear color filterelements, yellow color filter elements, etc.), and/or any other desiredcolor filter elements may be used in place of the red, green, and/orblue color filter elements. Similarly, different color filter elementsmay be substituted for the near-infrared light color filter elements ifdesired. For example, clear color filter elements or another type ofbroadband color filter element may be used in place of the near-infraredlight color filter elements if desired. The channel including thenear-infrared (and/or other desired color filter elements) may sometimesbe referred to as a high fidelity or high sensitivity channel.

The aforementioned embodiments may provide an image sensor with visiblelight and NIR sensitivities at a negligible increase in cost compared toa sensor with a Bayer color filter. The aforementioned embodimentsprovide an ecosystem friendly solution that allows straightforwardintegration. The aforementioned embodiments may have sufficientperformance in both indoor and outdoor lighting conditions.

In various embodiments, an image sensor may include an array of imagingpixels comprising visible light pixels and near-infrared light pixelsand an array of color filter elements that cover the array of imagingpixels. The imaging pixels may be arranged in a pattern, the pattern mayinclude a repeating 2×2 unit cell of pixel groups, and each pixel groupmay include a visible light pixel sub-group and a near-infrared lightpixel sub-group.

The visible light pixel sub-group of each pixel group may include firstand second adjacent visible light pixels. The near-infrared light pixelsub-group of each pixel group may include first and second adjacentnear-infrared light pixels. The 2×2 unit cell of pixel groups mayinclude first, second, third, and fourth pixel groups. The visible lightpixel sub-group of the first pixel group may include a green pixelsub-group, the visible light pixel sub-group of the second pixel groupmay include a red pixel sub-group, the visible light pixel sub-group ofthe third pixel group may include a blue pixel sub-group, and thevisible light pixel sub-group of the fourth pixel group may include agreen pixel sub-group.

The near-infrared light pixel sub-group of the first pixel group may beinterposed between the green pixel sub-group of the first group and thered pixel sub-group of the second pixel group and the near-infraredlight pixel sub-group of the third pixel group may be interposed betweenthe blue pixel sub-group of the third pixel group and the green pixelsub-group of the fourth pixel group. The red pixel sub-group of thesecond pixel group may be interposed between the near-infrared lightpixel sub-group of the first pixel group and the near-infrared lightpixel sub-group of the second pixel group and the green pixel sub-groupof the fourth pixel group may be interposed between the near-infraredlight pixel sub-group of the third pixel group and the near-infraredlight pixel sub-group of the fourth pixel group. The near-infrared lightpixel sub-group of the second pixel group may be interposed between thenear-infrared light pixel sub-group of the first pixel group and the redpixel sub-group of the second pixel group and the near-infrared lightpixel sub-group of the fourth pixel group may be interposed between thenear-infrared light pixel sub-group of the third pixel group and thegreen pixel sub-group of the fourth pixel group. The green pixelsub-group of the first pixel group may be interposed between thenear-infrared light pixel sub-group of the first group and the nearinfrared-light pixel sub-group of the second pixel group and thenear-infrared light pixel sub-group of the third pixel group may beinterposed between the blue pixel sub-group of the third pixel group andthe green pixel sub-group of the fourth pixel group.

The image sensor may also include a plurality of microlenses. Eachvisible light pixel sub-group may be covered by a single respectivemicrolens of the plurality of microlenses and each near-infrared lightpixel sub-group may be covered by a single respective microlens of theplurality of microlenses. The imaging pixels of at least one pixel groupmay be covered by a single microlens. The image sensor may also includeprocessing circuitry that processes signals from the pixel groups andoutputs a corresponding Bayer pattern to a processor for demosaicing.The processing circuitry may be configured to process signals from thevisible light pixel sub-group and signals from the near-infrared lightpixel sub-group to obtain a first processed set of image data and asecond processed set of image data for each pixel group of the repeating2×2 unit cell of pixel groups. The first processed set of image data maybe a set of monochrome image data and the second processed set of imagedata may be a set of Bayer image data. The processing circuitry may beconfigured to mix the set of monochrome image data with the set of Bayerimage data into a set of output data that includes both near-infraredlight information and visible light information and output the set ofoutput data to a processor for demosaicing

In various embodiments, an image sensor may include a plurality ofphotodiodes and a plurality of color filter elements arranged in apattern. Each photodiode of the plurality of photodiodes may be coveredby a respective color filter element of the plurality of color filterelements, the pattern may include a repeating 2×2 unit cell of groups,and each group may include a first sub-group of visible light colorfilter elements and a second sub-group of near-infrared light colorfilter elements.

The first sub-group of each group may include first and second adjacentvisible light color filter elements. The second sub-group of each groupmay include first and second adjacent near-infrared light color filterelements. The 2×2 unit cell of groups may include first and secondgroups above third and fourth groups, the first sub-group of the firstpixel group may include first and second adjacent green color filterelements, the first sub-group of the second group may include first andsecond adjacent red color filter elements, the first sub-group of thethird group may include first and second adjacent blue color filterelements, and the first sub-group of the fourth group may include firstand second adjacent green color filter elements.

In various embodiments, a method of processing signals from an array ofimaging pixels that includes visible light pixels and near-infraredlight pixels may include receiving signals from a 4×4 unit cell ofimaging pixels that includes first, second, third and fourth groups ofimaging pixels, for each group of imaging pixels, processing thereceived signals to obtain a single representative value for the groupof imaging pixels, and outputting the representative value for eachgroup of imaging pixels in a Bayer pattern. Each group of imaging pixelsmay include both visible light pixels and near-infrared light pixels.

Processing the received signals for the first group of imaging pixelsmay include determining the average of the signals from the visiblelight pixels and the near-infrared light pixels of the first group.Processing the received signals for the first group of imaging pixelsmay include determining the average of the signals from thenear-infrared light pixels of the first group. Processing the receivedsignals for the first group of imaging pixels may include determining afirst average of the signals from the near-infrared light pixels of thefirst group, determining a second average of the signals from thevisible light pixels of the first group, and subtracting the firstaverage from the second average. Processing the received signals for thefirst group of imaging pixels may include determining the average of thesignals from the visible light pixels of the first group.

The foregoing is merely illustrative of the principles of this inventionand various modifications can be made by those skilled in the art. Theforegoing embodiments may be implemented individually or in anycombination.

What is claimed is:
 1. An image sensor comprising: a plurality ofphotodiodes; and a plurality of color filter elements arranged in apattern, wherein each photodiode of the plurality of photodiodes iscovered by a respective color filter element of the plurality of colorfilter elements, wherein the pattern comprises a repeating 2×2 unit cellof groups, and wherein each group comprises at least a first colorfilter element of a first type that is surrounded by at least a firstcolor filter element of a second type that is different than the firsttype.
 2. The image sensor defined in claim 1, wherein the at least firstcolor filter element of the first type comprises a near-infrared lightcolor filter element.
 3. The image sensor defined in claim 2, whereinthe at least first color filter element of the second type comprises avisible light color filter element.
 4. The image sensor defined in claim3, wherein the 2×2 unit cell of groups has first, second, third, andfourth groups, wherein the visible light color filter element of thefirst group is a green color filter element, wherein the visible lightcolor filter element of the second group is a red color filter element,wherein the visible light color filter element of the third group is ablue color filter element, and wherein the visible light color filterelement of the fourth group is a green color filter element.
 5. Theimage sensor defined in claim 1, wherein the at least first color filterelement of the first type is a single near-infrared light color filterelement and wherein the at least first color filter element of thesecond type is a single visible light color filter element.
 6. The imagesensor defined in claim 1, wherein the at least first color filterelement of the first type comprises first and second near-infrared lightcolor filter elements.
 7. The image sensor defined in claim 6, whereinthe at least first color filter element of the second type comprisesfirst and second visible light color filter elements.
 8. The imagesensor defined in claim 1, wherein the at least first color filterelement of the first type comprises first, second, third, and fourthnear-infrared light color filter elements.
 9. The image sensor definedin claim 8, wherein the at least first color filter element of thesecond type comprises first, second, third, and fourth visible lightcolor filter elements.
 10. The image sensor defined in claim 1, whereineach group in the repeating 2×2 unit cell of groups is covered by asingle microlens.
 11. The image sensor defined in claim 1, wherein eachphotodiode is configured to obtain phase detection information.
 12. Animage sensor comprising: a plurality of pixels arranged in a pattern,wherein the pattern comprises a repeating 2×2 unit cell of groups,wherein each group comprises at least one near-infrared light pixel andat least one visible light pixel and wherein at least one pixel in eachgroup is configured to obtain phase detection information.
 13. The imagesensor defined in claim 12, wherein each group comprises first andsecond visible light pixels that are covered by a single microlens. 14.The image sensor defined in claim 12, wherein each group comprises firstand second near-infrared light pixels that are covered by a singlemicrolens.
 15. The image sensor defined in claim 12, wherein each groupcomprises first and second visible light pixels and first and secondnear-infrared light pixels that are covered by a single microlens. 16.The image sensor defined in claim 12, wherein each near-infrared lightpixel comprises a photodiode covered by a respective near-infrared lightcolor filter element and wherein each visible light pixel comprises aphotodiode covered by a respective visible light color filter element.17. An image sensor comprising: a plurality of photodiodes; a pluralityof color filter elements, wherein each photodiode of the plurality ofphotodiodes is covered by a respective color filter element of theplurality of color filter elements and wherein the plurality of colorfilter elements includes a visible light color filter element formedover a first photodiode and a near-infrared light color filter elementformed over a second photodiode; and a microlens that covers the firstand second photodiodes.
 18. The image sensor defined in claim 17,wherein the plurality of color filter elements includes an additionalvisible light color filter element formed over a third photodiode and anadditional near-infrared light color filter element formed over a fourthphotodiode and wherein the microlens covers the first, second, third,and fourth photodiodes.
 19. The image sensor defined in claim 17,wherein the visible light color filter element is a first visible lightcolor filter element, wherein the plurality of color filter elementsincludes second, third, and fourth visible light color filter elementsformed over respective third, fourth, and fifth photodiodes, wherein thenear-infrared light color filter element is a first near-infrared lightcolor filter element, and wherein the plurality of color filter elementsincludes second, third, and fourth near-infrared light color filterelements formed over respective sixth, seventh, and eighth photodiodes.20. The image sensor defined in claim 19, wherein the microlens coversthe first, second, third, fourth, fifth, sixth, seventh, and eightphotodiodes.